Machine-learning based gesture recognition using multiple sensors

ABSTRACT

A device implementing a system for machine-learning based gesture recognition includes at least one processor configured to, receive, from a first sensor of the device, first sensor output of a first type, and receive, from a second sensor of the device, second sensor output of a second type that differs from the first type. The at least one processor is further configured to provide the first sensor output and the second sensor output as inputs to a machine learning model, the machine learning model having been trained to output a predicted gesture based on sensor output of the first type and sensor output of the second type. The at least one processor is further configured to determine the predicted gesture based on an output from the machine learning model, and to perform, in response to determining the predicted gesture, a predetermined action on the device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.16/937,481, entitled “Machine-Learning Based Gesture Recognition UsingMultiple Sensors”, filed on Jul. 23, 2020, which claims the benefit ofU.S. Provisional Patent Application Ser. No. 62/933,232, entitled“Machine-Learning Based Gesture Recognition Using Multiple Sensors,”filed on Nov. 8, 2019, which is hereby incorporated by reference in itsentirety for all purposes.

TECHNICAL FIELD

The present description relates generally to gesture recognition,including machine-learning based gesture recognition.

BACKGROUND

The present disclosure relates generally to electronic devices and inparticular to detecting gestures made by a user wearing or otherwiseoperating an electronic device.

BRIEF DESCRIPTION OF THE DRAWINGS

Certain features of the subject technology are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of thesubject technology are set forth in the following figures.

FIG. 1 illustrates an example network environment for providingmachine-learning based gesture recognition in accordance with one ormore implementations.

FIG. 2 illustrates an example device that may implement a system formachine-learning based gesture recognition in accordance with one ormore implementations.

FIG. 3 illustrates an example architecture, that may be implemented byan electronic device, for machine-learning based gesture recognition inaccordance with one or more implementations.

FIGS. 4A-4B illustrate example diagrams of respective sensor outputs ofan electronic device that may indicate a gesture in accordance with oneor more implementations.

FIG. 5 illustrates a flow diagram of example process formachine-learning based gesture recognition in accordance with one ormore implementations.

FIG. 6 illustrates an example diagram of a binary label for sensor datathat may indicate a gesture in accordance with one or moreimplementations.

FIG. 7 illustrates an example of smooth labels for a gesture that may beindicated by sensor data in accordance with one or more implementations.

FIG. 8 illustrates additional examples of smooth labels for a gesturethat may be indicated by sensor data in accordance with one or moreimplementations.

FIG. 9 illustrates a flow diagram of another example process formachine-learning based gesture recognition in accordance with one ormore implementations.

FIG. 10 illustrates an example electronic system with which aspects ofthe subject technology may be implemented in accordance with one or moreimplementations.

DETAILED DESCRIPTION

The detailed description set forth below is intended as a description ofvarious configurations of the subject technology and is not intended torepresent the only configurations in which the subject technology can bepracticed. The appended drawings are incorporated herein and constitutea part of the detailed description. The detailed description includesspecific details for the purpose of providing a thorough understandingof the subject technology. However, the subject technology is notlimited to the specific details set forth herein and can be practicedusing one or more other implementations. In one or more implementations,structures and components are shown in block diagram form in order toavoid obscuring the concepts of the subject technology.

Electronic devices, such as smartwatches, may be configured to includevarious sensors. For example, a smartwatch may be equipped with one ormore biosignal sensors (e.g., a photoplethysmogram (PPG) sensor), aswell as other types of sensors (e.g., a motion sensor, an opticalsensor, an audio sensor and the like). The various sensors may workindependently and/or in conjunction with each other to perform one ormore tasks, such as detecting device position, environmental conditions,user biological conditions and the like.

In some cases, a user may wish to use touch input (e.g., on atouchscreen of the electronic device) to perform an action.Alternatively or in addition, it may be desirable for a user to performa gesture without having to rely on touch input. For example, a user maywish for the electronic device to perform a particular action based on agesture performed by the same hand wearing the smartwatch.

The subject technology provides for detecting user gestures by utilizingoutputs received via one or more sensors of the electronic device. Forexample, the electronic device may receive respective outputs from firstsensor(s) (e.g., biosignal sensor(s)) and second sensor(s) (e.g.,non-biosignal sensor(s)). The outputs may be provided as input to amachine learning model implemented on the electronic device, which hadbeen trained based on outputs from various sensors, in order to predicta user gesture. Based on the predicted gesture, the electronic devicemay perform a particular action (e.g., changing a user interface). Inone or more implementations, the machine learning model may be trainedbased on a general population of users, rather than a specific singleuser. In this manner, the model can be re-used across multiple differentusers even without a priori knowledge of any particular characteristicsof the individual users. In one or more implementations, a model trainedon a general population of users can later be tuned or personalized fora specific user.

FIG. 1 illustrates an example network environment 100 for providingmachine-learning based gesture recognition in accordance with one ormore implementations. Not all of the depicted components may be used inall implementations, however, and one or more implementations mayinclude additional or different components than those shown in thefigure. Variations in the arrangement and type of the components may bemade without departing from the spirit or scope of the claims as setforth herein. Additional components, different components, or fewercomponents may be provided.

The network environment 100 includes electronic devices 102, 103 and 104(hereinafter 102-104), a network 106 and a server 108. The network 106may communicatively (directly or indirectly) couple, for example, anytwo or more of the electronic devices 102-104 and the server 108. In oneor more implementations, the network 106 may be an interconnectednetwork of devices that may include, and/or may be communicativelycoupled to, the Internet. For explanatory purposes, the networkenvironment 100 is illustrated in FIG. 1 as including electronic devices102-104 and a single server 108; however, the network environment 100may include any number of electronic devices and any number of servers.

One or more of the electronic devices 102-104 may be, for example, aportable computing device such as a laptop computer, a smartphone, asmart speaker, a peripheral device (e.g., a digital camera, headphones),a tablet device, a wearable device such as a smartwatch, a band, and thelike, or any other appropriate device that includes, for example, one ormore wireless interfaces, such as WLAN radios, cellular radios,Bluetooth radios, Zigbee radios, near field communication (NFC) radios,and/or other wireless radios. In FIG. 1, by way of example, theelectronic device 102 is depicted as a smartwatch, the electronic device103 is depicted as a laptop computer, and the electronic device 104 isdepicted as a smartphone.

As is discussed further below, each of the electronic devices 102-104may include one or more sensors that can be used and/or repurposed todetect input received from a user. Each of the electronic devices102-104 may be, and/or may include all or part of, the device discussedbelow with respect to FIG. 2, and/or the electronic system discussedbelow with respect to FIG. 10.

The server 108 may be, and/or may include all or part of the electronicsystem discussed below with respect to FIG. 10. The server 108 mayinclude one or more servers, such as a cloud of servers. For explanatorypurposes, a single server 108 is shown and discussed with respect tovarious operations. However, these and other operations discussed hereinmay be performed by one or more servers, and each different operationmay be performed by the same or different servers. In one or moreimplementations, one or more of the electronic devices 102-104 mayimplement the subject system independent of the network 106 and/orindependent of the server 108.

FIG. 2 illustrates an example device that may implement a system formachine-learning based gesture recognition in accordance with one ormore implementations. For explanatory purposes, FIG. 2 is primarilydescribed herein with reference to the electronic device 102 of FIG. 1.Not all of the depicted components may be used in all implementations,however, and one or more implementations may include additional ordifferent components than those shown in the figure. Variations in thearrangement and type of the components may be made without departingfrom the spirit or scope of the claims as set forth herein. Additionalcomponents, different components, or fewer components may be provided.

The electronic device 102 may include a host processor 202, a memory204, one or more biosignal sensor(s) 206, one or more non-biosignalsensor(s) 208, and a communication interface 210. The host processor 202may include suitable logic, circuitry, and/or code that enableprocessing data and/or controlling operations of the electronic device102. In this regard, the host processor 202 may be enabled to providecontrol signals to various other components of the electronic device102. The host processor 202 may also control transfers of data betweenvarious portions of the electronic device 102. The host processor 202may further implement an operating system or may otherwise execute codeto manage operations of the electronic device 102.

The memory 204 may include suitable logic, circuitry, and/or code thatenable storage of various types of information such as received data,generated data, code, and/or configuration information. The memory 204may include, for example, random access memory (RAM), read-only memory(ROM), flash, and/or magnetic storage.

In one or more implementations, the biosignal sensor(s) 206 may includeone or more sensors configured to measure biosignals. For example, thebiosignal sensor(s) 206 may correspond to a photoplethysmography (PPG)PPG sensor configured to detect blood volume changes in microvascularbed of tissue of a user (e.g., where the user is wearing the electronicdevice 102 on his/her body, such as his/her wrist). The PPG sensor mayinclude one or more light-emitting diodes (LEDs) which emit light and aphotodiode/photodetector (PD) which detects reflected light (e.g., lightreflected from the wrist tissue). The biosignal sensor(s) 206 are notlimited to a PPG sensor, and may additionally or alternativelycorrespond to one or more of: an electroencephalogram (EEG) sensor, anelectrocardiogram (ECG) sensor, an electromyogram (EMG) sensor, amechanomyogram (MMG) sensor, an electrooculography (EOG) sensor, agalvanic skin response (GSR) sensor, a magnetoencephalogram (MEG) sensorand/or other suitable sensor(s) configured to measure biosignals.

In one or more implementations, the non-biosignal sensor(s) 208 mayinclude one or more sensors for detecting device motion, sound, light,wind and/or other environmental conditions. For example, thenon-biosignal sensor(s) 208 may include one or more of: an accelerometerfor detecting device acceleration, an audio sensor (e.g., microphone)for detecting sound, an optical sensor for detecting light, and/or othersuitable sensor(s) configured to output signals indicating device stateand/or environmental conditions.

As discussed further below with respect to FIGS. 3-9, one or more of theelectronic devices 102-104 may be configured to output a predictedgesture based on output provided by the biosignal sensor(s) 206 and/oroutput by the non-biosignal sensor(s) 208 (e.g., corresponding to inputsdetected by the biosignal sensor(s) 206 and the non-biosignal sensor(s)208).

The communication interface 210 may include suitable logic, circuitry,and/or code that enables wired or wireless communication, such asbetween the electronic device 102 and other device(s). The communicationinterface 210 may include, for example, one or more of a Bluetoothcommunication interface, an NFC interface, a Zigbee communicationinterface, a WLAN communication interface, a USB communicationinterface, or generally any communication interface.

In one or more implementations, one or more of the host processor 202,the memory 204, the biosignal sensor(s) 206, the non-biosignal sensor(s)208, the communication interface 210, and/or one or more portionsthereof, may be implemented in software (e.g., subroutines and code),may be implemented in hardware (e.g., an Application Specific IntegratedCircuit (ASIC), a Field Programmable Gate Array (FPGA), a ProgrammableLogic Device (PLD), a controller, a state machine, gated logic, discretehardware components, or any other suitable devices) and/or a combinationof both.

FIG. 3 illustrates an example architecture 300, that may be implementedby an electronic device, for machine-learning based gesture recognitionin accordance with one or more implementations. Not all of the depictedcomponents may be used in all implementations, however, and one or moreimplementations may include additional or different components thanthose shown in the figure. Variations in the arrangement and type of thecomponents may be made without departing from the spirit or scope of theclaims as set forth herein. Additional components, different components,or fewer components may be provided.

As illustrated, the gesture prediction engine 302 includes a machinelearning model 304. The machine learning model 304, in an example, isimplemented as a neural network (NN) model that is configured to detecta gesture using such sensor inputs over time. As discussed herein, aneural network (NN) is a computing model that uses a collection ofconnected nodes to process input data based on machine learningtechniques. Neural networks are referred to as networks because they maybe represented by connecting together different operations. A model of aNN (e.g., feedforward neural network) may be represented as a graphrepresenting how the operations are connected together from an inputlayer, through one or more hidden layers, and finally to an outputlayer, with each layer including one or more nodes, and where differentlayers perform different types of operations on respective input.

In one or more implementations, the machine learning model 304 isimplemented as a convolutional neural network (CNN). As discussedherein, a CNN refers to a particular type of neural network, but usesdifferent types of layers made up of nodes existing in three dimensionswhere the dimensions may change between layers. In a CNN, a node in alayer may only be connected to a subset of the nodes in a previouslayer. The final output layer may be fully connected and be sizedaccording to the number of classifiers. A CNN may include variouscombinations, and in some instances, multiples of each, and orders ofthe following types of layers: the input layer, convolutional layers,pooling layers, rectified linear unit layers (ReLU), and fully connectedlayers. Part of the operations performed by a convolutional neuralnetwork includes taking a set of filters (or kernels) that are iteratedover input data based on one or more parameters.

In an example, convolutional layers read input data (e.g., a 3D inputvolume corresponding to sensor output data, a 2D representation ofsensor output data, or a 1D representation of sensor output data), usinga kernel that reads in small segments at a time and steps across theentire input field. Each read can result in an input that is projectedonto a filter map and represents an internal interpretation of theinput. A CNN such as the machine learning model 304, as discussedherein, can be applied to human activity recognition data (e.g., sensordata corresponding to motion or movement) where the CNN model learns tomap a given window of signal data to an activity (e.g., gesture and/orportion of a gesture) where the model reads across each window of dataand prepares an internal representation of the window.

The machine learning model 304 may be configured to receive output fromone, two or more than two sensors (e.g., at least one biosignal sensorand/or at least one non-biosignal sensor) as input. As shown in theexample of FIG. 3, the machine learning model 304 receives firstbiosignal sensor output 306 to M^(th) biosignal sensor output 308, aswell as first non-biosignal sensor output 310 to N^(th) non-biosignalsensor output 312, as input.

The first biosignal sensor output 306 to the M^(th) biosignal sensoroutput 308 includes output from one or more of the biosignal sensor(s)206. As noted above, the biosignal sensor(s) 206 may correspond to a PPGsensor (e.g., for detecting blood volume changes) and/or other types ofsensor(s) configured to output biosignals. Moreover, the firstnon-biosignal sensor output 310 to N^(th) non-biosignal sensor output312 includes output from one or more of the non-biosignal sensor(s) 208.As noted above, the non-biosignal sensor(s) 208 may correspond to one ormore of an accelerometer, an optical sensor, an audio sensor (e.g., amicrophone) and/or other types of sensor(s) configured to output signalsindicating device state and/or environmental conditions.

In one or more implementations, one or more of the sensor outputs306-312 may correspond to a window of time (e.g., 0.5 seconds, 0.1seconds, or any window of time) in which sensor data was collected bythe respective sensor. Moreover, the sensor outputs 306-312 may befiltered and/or pre-processed (e.g., normalized) before being providedas inputs to the machine learning model 304.

In one or more implementations, the sensor outputs 306-312 may be usedto indicate a gesture performed by the user. For example, the gesturemay correspond to a single-handed gesture performed by the same handthat is coupled to (e.g., wearing) the electronic device 102. Thegesture may correspond to a static gesture (e.g., a specific type ofhand/finger positioning that is held for a predefined time period)and/or a dynamic gesture (e.g., a motion-based gesture performed over apredefined time period). Moreover, the gesture may correspond to afinger-based gesture (e.g., in which the fingers move and/or arepositioned in a specific manner), a wrist-based gesture (e.g., in whichthe wrist moves and/or is positioned in a specific manner) and/or acombination of a finger-based and wrist-based gesture. In one or moreimplementations, the gesture may correspond to a gesture performed on ahorizontal and/or vertical surface, such as, for example, a table, awall, a floor, and/or another hand.

Moreover, the sensor outputs 306-312 may individually and/orcollectively be used by the machine learning model 304 to indicate aspecific type of user gesture. As noted above, one or more of thebiosignal sensor(s) 206 may correspond to a PPG sensor configured todetect blood volume changes. For example, variations in blood volume mayindicate different user gestures (e.g., where particular blood volumechanges map to respective types of user gestures). As further notedabove, the machine learning model 304 may receive non-biological signaloutput (e.g., the non-biosignal sensor outputs 310-312), which may beused in conjunction with the biosignal sensor output(s) 306-308, assupplemental information predict the specific gesture. For example, thenon-biosignal sensor outputs 310-312 may indicate false positives forgesture predictions otherwise indicated by the biosignal sensor outputs306-308.

The machine learning model 304 (e.g., a CNN) may have been trained(e.g., pre-trained) on different device(s) (e.g., one or moresmartwatches other than the electronic device 102) based on sensoroutput data prior to being deployed on the electronic device 102. Thesensor output data for training may correspond to output from one ormore biosignal sensor(s) (e.g., similar to the biosignal sensor(s) 206)and/or from one or more non-biosignal sensors (e.g., similar to thenon-biosignal sensor(s) 208). In one or more implementations, themachine learning model 304 may have been trained across multiple users,for example, who provided different types of gestures while wearing adevice (e.g., another smartwatch with biosignal and/or non-biosignalsensor(s)) and confirmed the gestures (e.g., via a training userinterface) as part of a training process. In this manner, the machinelearning model may be used, in one or more implementations, to predictgestures across a general population of users, rather than one specificuser.

After the machine learning model 304 has been trained, the machinelearning model 304 may generate a set of output predictionscorresponding to gesture prediction(s) 314. After the predictions aregenerated, a policy may be applied to the predictions to determinewhether to indicate an action for the electronic device 102 to perform,which is discussed in more detail with respect to FIGS. 4A-4B.

FIGS. 4A-4B illustrate example diagrams of respective sensor outputs ofan electronic device that may indicate a gesture in accordance with oneor more implementations. For explanatory purposes, FIGS. 4A-4B areprimarily described herein with reference to the electronic device 102of FIG. 1. However, FIGS. 4A-4B are not limited to the electronic device102 of FIG. 1, and one or more other components and/or other suitabledevices (e.g., any of the electronic device 102-104) may be usedinstead.

FIG. 4A illustrates an example in which the electronic device 102includes a PPG sensor 402. The PPG sensor 402 includes one or more lightsources 404A-404B (e.g., LEDs) configured to emit light. For example,the light source 404A may emit light corresponding to a first frequency(e.g., green light) and the light source 404B may emit lightcorresponding to a second frequency (e.g., another color such as brownand/or infrared light). The PPG sensor 402 further includes one or morephotodiodes 406A-406B configured to detect reflected light (e.g., lightreflected from wrist tissue of the user, based on light emitted by thelight sources 404A-404B). The PPG sensor 402 may be configured toaverage or otherwise process the output from the photodiodes 406A-406Bto provide output (e.g., the first biosignal sensor output 306)corresponding to blood volume changes. The electronic device 102 mayfurther include an accelerometer (not shown) configured to detect deviceacceleration.

FIG. 4B illustrates example timing diagrams 408-410 of respective sensoroutputs of the electronic device 102, together with respectiveconfidence levels corresponding to a particular user gesture. Forexample, the timing diagrams 408-410 may indicate the confidence of afist-clinch gesture.

The timing diagram 408 illustrates sensor output of the PPG sensor 402of the electronic device 102 together with confidence output (e.g.,based on the machine learning model 304) that the sensor outputcorresponds to a particular user gesture (e.g., fist-clinch). Sensoroutputs 412A-412B correspond to reflected light detected by thephotodiodes 406A-406B based on light emitted by the light sources404A-404B. While FIG. 4B illustrates the example of a light source whichis green, the PPG may include alternative and/or additional lightsources (e.g., other colors such as brown, infrared light, and thelike). The sensor output 414 corresponds to an average of the sensoroutputs 412A-412B. Moreover, the confidence output 416 (e.g., based onthe machine learning model 304) indicates that the sensor outputcorresponds to a particular user gesture.

In one or more implementations, the timing diagram 410 illustratessensor output of an accelerometer of the electronic device 102 togetherwith confidence output (e.g., by the machine learning model 304) thatthe sensor output corresponds to a particular user gesture (e.g.,fist-clinch). The sensor output 418 corresponds to detected acceleration(e.g., based on device movement). Moreover, the confidence value 420indicates the calculated confidence (e.g., based on the machine learningmodel 304) that the sensor output indicates a particular user gesture.

In one or more implementations, the machine learning model 304 may beconfigured to provide gesture prediction(s) (e.g., corresponding togesture prediction(s) 314) on a periodic basis (e.g., 10 predictions persecond, or some other amount of predictions per time period) based onthe aforementioned sensor output data which is visually shown in thetiming diagrams 408-410. While FIGS. 4A-4B are described with respect tothe example of a fist-clinch gesture, the machine learning model 304 maybe configured to provide gesture predictions with respect to multipledifferent types of gestures (e.g., static and/or dynamic finger-basedgestures, static and/or dynamic wrist-based gestures).

In one or more implementations, as mentioned above, the machine learningmodel 304 may utilize a policy to determine a prediction output. Asreferred to herein, a policy can correspond to a function thatdetermines a mapping of a particular input (e.g., sensor output data) toa corresponding action (e.g., providing a respective prediction). Forexample, the machine learning model 304 may utilize sensor output datacorresponding to a particular gesture to make a classification, and thepolicy can determine an average of a number of previous predictions(e.g., 5 previous predictions). The machine learning model 304 may takethe previous predictions over a window of time, and when the average ofthese predictions exceeds a particular threshold, the machine learningmodel 304 can indicate a particular action (e.g., updating a userinterface) for the electronic device 102 to initiate. In one or moreimplementations, the policy may be applied to an output of the machinelearning model 304.

In one or more implementations, a state machine may be utilized tofurther refine the predictions output by the machine learning model 304(e.g. based on previous predictions over a window of time). For example,the state machine may include one or more transitional states between agesture detected and a gesture not detected, such as start of gesturedetected, middle of gesture detected, end of gesture detected, and thelike.

FIG. 5 illustrates a flow diagram of example process formachine-learning based gesture recognition in accordance with one ormore implementations. For explanatory purposes, the process 500 isprimarily described herein with reference to the electronic device 102of FIG. 1. However, the process 500 is not limited to the electronicdevice 102 of FIG. 1, and one or more blocks (or operations) of theprocess 500 may be performed by one or more other components and othersuitable devices (e.g., any of the electronic devices 102-104). Furtherfor explanatory purposes, the blocks of the process 500 are describedherein as occurring in serial, or linearly. However, multiple blocks ofthe process 500 may occur in parallel. In addition, the blocks of theprocess 500 need not be performed in the order shown and/or one or moreblocks of the process 500 need not be performed and/or can be replacedby other operations.

The electronic device 102 receives, from one or more of the biosignalsensor(s) 206, first sensor output of a first type (502). The biosignalsensor(s) 206 of the device may be a photoplethysmography (PPG) sensor.The PPG sensor may include at least one of an infrared light source or acolor light source. In one or more implementations, the first sensoroutput may indicate a change in blood flow.

The electronic device 102 receives, from one or more of thenon-biosignal sensor(s) 208, second sensor output of a second type thatdiffers from the first type (504). The non-biosignal sensor(s) 208 maybe an accelerometer and/or a microphone. At least one of receiving thefirst sensor output or receiving the second sensor output may be basedon a determination that the device is in a gesture detection mode.

The electronic device 102 provides the first sensor output and thesecond sensor output as inputs to a machine learning model, the machinelearning model having been trained to output a predicted gesture basedon sensor output of the first type and sensor output of the second type(506). The machine learning model may have been trained across multipleusers.

The electronic device 102 determines the predicted gesture based on anoutput from the machine learning model (508). The predicted gesture maybe at least one of a finger-based gesture, or a wrist-based gesture. Forexample, the finger-based gesture may be at least one of a finger pinchgesture (e.g., touching two fingers together), a double pinch or othermultiple pinch (e.g., touching two fingers together multiple times witha separation of the two fingers in between the multiple touches), afist-clinch gesture (e.g., holding one or more (or all) fingers and/orthumb in the form of a fist), and/or a double-clinch gesture or othermultiple clinch gesture. For example, the wrist-based gesture may be atleast one of a knock gesture or a double knock gesture.

The electronic device 102 performs, in response to determining thepredicted gesture, a predetermined action on the device (510). Thepredetermined action may correspond to changing a user interface on thedevice. These predetermined actions can provide, in one or moreimplementations, gesture-powered switch control (e.g., foraccessibility) for an electronic device. For example, gesture-poweredswitch control can allow a user to navigate an operating system of asmartwatch using only the watch-wearing arm. Gesture-powered switchcontrol can include operating a user interface (UI) element that ishighlighted by a selector, by performing a gesture while the UI elementis highlighted by the selector.

The predetermined actions can also enable users to set shortcuts thatare accessed uniquely by corresponding gestures. For example, shortcutshaving associated gestures can be provided automatically by context(e.g., including shortcuts and corresponding gestures for interactingwith a media player application, shortcuts and corresponding gesturesfor interacting with a workout application, and/or shortcuts andcorresponding gestures for interacting with any other application).

The predetermined actions can also include providing instructions to acompanion device (e.g., a mobile phone, a laptop, a tablet, anotherwearable device, etc. that is communicatively coupled to a wearablegesture-detecting device such as a smartwatch), to enable gesture-basedcontrol of the companion device. For example, a predetermined actionresponsive to a predicted gesture can include sending gestureinformation or an instruction to a companion device that is playingmedia (e.g., audio or video) to skip to a next or previous track orchapter, pause or restart the media, or perform other medial controloperations at the companion device. As another example, a predeterminedaction responsive to a predicted gesture can include sending gestureinformation or an instruction to a companion device that is displaying abrowser or a document to scroll or perform other control of the browseror document. As another example, a predetermined action responsive to apredicted gesture can include sending gesture information or aninstruction to a companion device that is running an augmented realityapplication or a virtual reality application, for input to or control ofthe application.

In one or more implementations, a machine learning model for gestureprediction and/or identification can include a portion that initiallypredicts whether the model should be in a gesture detection mode. Forexample, the machine learning models described above in connection withFIGS. 1-5 can include a prediction head in the neural network thatpredicts whether the remaining portions of the model (or a separatemodel) should start model prediction or not. This additional predictionhead can be helpful, for example, to save energy and computation time(e.g., to help allow gesture detection to constantly run in thebackground even on devices with limited power supplies such asbatteries). In this way, machine learning models can be provided forwhich the data cube does not have to perform operations all the way downto the end of the network if the additional gesture-detection headindicates that a gesture is not occurring. The prediction head fordetermining whether the model proceeds to a gesture prediction mode canbe trained in a common training operation with other portions of themodel, or trained separately from a separate gesture prediction model.

In one or more implementations, a machine learning model (e.g., machinelearning model 304) may generate, for the data in a data buffer (e.g., adata buffer storing sensor data from a particular window of time), aconfidence output (see, e.g., confidence output 416) or a confidencevalue (see, e.g., confidence value 420) that indicates whether the datain the data buffer indicates a particular gesture being performed by auser. In one or more implementations, the machine learning model mayalso be arranged and trained to generate labels for whether a gesture isoccurring (e.g., a binary gesture/no-gesture label, a start label and anend label, and/or a smoothly continuous gesture label and/or no-gesturelabel). FIG. 6 illustrates an example in which the machine learningmodel generates a binary gesture/no-gesture label 600.

As shown in FIG. 6, for sensor data 602 that includes a portion 604corresponding to a gesture performed by a user, a gesture/no-gesturelabel 600 can have a value of, e.g., one when a gesture is beingperformed and a value of, e.g., zero, when no gesture is beingperformed. In one or more implementations, the times at which thegesture/no-gesture label 600 transitions from low to high and from highto low can indicate a start time 608 and an end time 610 of a gesture(G) having a gesture duration 606.

As indicated in FIG. 6, a machine learning model such as machinelearning model 304 can be run on sensor data collected within a window(W) having a window duration 612. For example, sensor data 602 from asliding window (W) having a window duration 612, can be loaded into abuffer that is accessible by the machine learning model at each ofseveral times, to provide the sensor data from that window as input tothe machine learning model. In the example of FIG. 6, the model may beexecuted for a window 614 that is entirely before the gesture, windows616 and 618 that are partially overlapping with the gesture includingthe beginning of the gesture, a window 620 that includes the entiregesture, and one or more windows such as window 622 that is partiallyoverlapping with the gesture including the end of the gesture. For eachof windows 614, 616, 618 620, 622, etc., the model may generate and/oroutput a binary gesture/no-gesture label 600 indicating whether agesture is occurring within the window, and a prediction of whichgesture is occurring within the window. In one or more implementations,the labels and/or predictions corresponding to multiple windows can becombined to determine a final start time 608, a final end time 610,and/or a final predicted gesture that occurred between the final starttime and the final end time.

For example, a machine learning model such as machine learning model 304may be provided that includes a multi-tasking network head (e.g., at theend of model) to predict the start and end time of the gesture based onthe data in the data buffer (e.g., even for windows such as windows 616,618, or 622 of FIG. 6 in which the start time 608 and/or end time 610 ofthe gesture may not necessarily be inside the data buffer). For example,the model may be arranged and trained to predict when the gestureactually started and when the gesture is going to end based on thepartial information from the gesture that is present in the data bufferat any given time. For example, the machine learning model may includeparallel gesture-classification and region-of-interest (ROI) regressionheads at the end of the model, the outputs of which can be concatenatedfor output from the model. The gesture-classification head may generate,for example, a prediction of which gesture is being performed. The ROIregression head may generate, for example, the gesture and/or no-gesturelabels for determining the start time and the end time of the gesturebeing classified, and/or generate the predicted start time and/orpredicted end time based on the generated labels.

Multiple gesture start and end timestamps from the rolling predictionwindows can be combined to predict the final start and end times for thepredicted gesture. For example, aggregated predicted start and endindices corresponding to outputs based on multiple data buffers outputscan be used to identify the start and end indices of a complete gesture,since the multiple data buffers together include the data from the wholegesture duration 606 of the gesture.

In one or more implementations, combining predicted start times and endtimes for multiple sampling windows can include, for each samplingwindow, determining a region of interest within that window, convertingthe region of interest into indices of interest (IOIs) in buffercoordinates, translating the IOIs in buffer coordinates to IOIsrespective to a common gating period, aggregating the translated IOIsinto an aggregated IOI, and translating the aggregated IOI into indexcoordinates with an origin at a time equal to zero.

In one or more implementations, a machine learning model that performsmultistep prediction during the gesture in this way (e.g., instead ofassigning a single prediction to a data buffer), can provide predictionsof multiple labels for different parts of the buffer. In this way, themachine learning model can transform a sequence of data in the databuffer into a sequence of labels corresponding to different parts of thebuffer.

Although multistep gesture prediction can be performed using a binarygesture/no-gesture label 600 as in the example of FIG. 6, the binarygesture/no-gesture labeling of FIG. 6 may not account for noise in thetraining data (e.g., due to noisy training labels for the start and endtimes for a training gesture) and/or noise in the sensor data (e.g., dueto user variations in how a gesture is performed). In order to provide amore robust and accurate model, a machine learning model such as machinelearning model 304 may be arranged and trained to generate smoothedlabels for identifying the start and/or end of a gesture.

FIG. 7 illustrates an example of smoothed labels for gesture predictionthat can be generated using a machine learning model such as machinelearning model 304, in one or more implementations. In the example ofFIG. 7, the start time 608 and end time 610 of a gesture can bedetermined using a gesture label 700 and a no-gesture label 702 that caneach have multiple values (e.g., discrete or continuous values) betweena minimum value (e.g., zero) and a maximum value (e.g., one). As shown,for each of windows 714, 716, 718, 720, 722, etc., the model maygenerate and/or output both a gesture label 700 (e.g., indicating aprobability that a gesture is occurring in that window) and a no-gesturelabel 702 (e.g., indicating a probability that no gesture is occurringin that window), and a prediction (e.g., classification) of whichgesture is occurring within the window.

As indicated in FIG. 7, for a window 714 that does not include anysensor data associated with a gesture, a gesture label 700 may have aminimum value such as a value of zero, and a no-gesture label 702 mayhave a maximum value such as a value of one. As the rolling or movingwindow begins to include the gesture, the gesture label 700 begins to(e.g., smoothly) rise and the no-gesture label 702 begins to (e.g.,smoothly) decrease until, in window 720 which entirely overlaps thegesture, the gesture label 700 reaches a maximum value (e.g., one) andthe no-gesture label 702 reaches a minimum value (e.g., zero). As therolling or moving window begins to include sensor data obtained afterthe gesture is complete, the gesture label 700 begins to (e.g.,smoothly) decrease and the no-gesture label 702 begins to (e.g.,smoothly) rise until, when the window no longer overlaps any portion ofthe gesture, the gesture label 700 reaches minimum value (e.g., zero)and the no-gesture label 702 reaches a maximum value (e.g., one).

A machine learning model such as machine learning model 304 thatpredicts the smooth labels of FIG. 7 (e.g., instead of binary label ofFIG. 6) can indicate how much of a gesture has been seen by the model,and which gesture has been seen. The model output of a machine learningmodel that predicts the smooth labels of FIG. 7 can output not only aprobability score, but also a prediction (e.g., for each window) of howfar the current data in the data buffer extends into the gesture beingperformed. For example, the values of smooth labels such as the gesturelabel 700 and the no-gesture label 702 can be generated based on agesture interval and depending on the size of data buffer in the model,to allow the labels to reflect how much of the gesture overlapped withthe data buffer. A thresholding strategy can be applied on top of thesmooth predicted labels to determine when the data buffer is mostly orcompletely inside a gesture being performed (e.g., when the gesturelabel 700 is above a threshold such as 0.9 and/or when the no-gesturelabel 702 is below a threshold such as 0.1).

As in the case of binary gesture/no-gesture label 600, in one or moreimplementations, the model outputs corresponding to the multiple windows714, 716, 718, 720, 722, etc., can be combined to determine a finalstart time 608, a final end time 610, and a final predicted gesture thatoccurred between the final start time and the final end time. In variousimplementations, the predicted gesture that was previously generatedwith the highest gesture label 700 and/or the lowest no-gesture label702 can be used as the final predicted gesture, or the final gestureprediction can be generated after the final start time and final endtime have been determined (e.g., by re-running the gesture predictionwith the data between the final start time and final end time and thusincluding the entire gesture).

In one or more implementations, the buffer size for the input data tothe machine learning model can be adjusted for the final gestureprediction, based on the final start time 608 and the final end time610. For example, for a gesture having a gesture duration of 100milliseconds (ms), a buffer size may be reduced from 1 second to 200 msfor the final gesture prediction (e.g., to avoid including unnecessaryand potential confusing data in the buffer). In another example, for agesture having a gesture duration of 1.3 seconds, a default 1 secondbuffer size can be increased (e.g., to ensure the sensor data for theentire gesture is included in the buffer) for the final gestureprediction. In one or more implementations, when smooth labels such asthe gesture label 700 and the no-gesture label 702 of FIG. 7 are used,instead of cross entropy and softmax functions at the output layer ofthe machine learning model, binary cross entropy and sigmoid functionscan be applied.

Predicting the start and end times of the gesture can be helpful forproviding a machine learning model that can detect multi-movementgestures. For example, in order to provide a machine learning model thatcan predict and/or detect both a single pinch and a double pinch, orboth a single clinch and a double clinch, the predicted start and endtimes can help avoid excluding data corresponding to the second pinch orthe second clinch in a double gesture.

It should be appreciated that the gesture label 700 and the no-gesturelabel 702 shown in FIG. 7, which are linearly increasing or decreasingbetween minimum and maximum values, are merely illustrative. FIG. 8illustrates other smooth gesture labels 700 and no-gesture labels 702that can be used. For example, FIG. 8 illustrates sigmoid andexponential gesture labels 700 and no-gesture labels 702. Providingsmooth gesture labels can also include providing an additional scoreindicating a goodness of a particular window (e.g., a window proposalscore), and/or using multi-task learning (e.g., using an additionalregressor to indicate which part of a gesture is within a particularprediction window). Although the smooth labeling of FIGS. 7 and 8 aredescribed in the context of gesture detection and/or gesture prediction,it should be appreciated that such smooth labeling of start times andend times in sensor data can be applied to incorporate statisticaluncertainty into the labels for other data for detecting occurrencesthat are limited in time (e.g., for any sensor data for which boundarydetection in time-series data is desired so that action can be takenbased on the sensor data within or near the boundary or boundaries).

In the examples of FIGS. 6 and 7, the windows 614-622 and 714-722 areused to sample the data uniformly in overlapping sliding windows oftime. It should also be appreciated that, in one or moreimplementations, sampling of data during training of the machinelearning model and/or during client use of the machine learning modelmay be performed pseudo-randomly (e.g., using windows of a common widththat are centered at pseudo-random times around the gesture, rather thancentered at uniformly progressing times before, during, and after thegesture). Evaluation of the model can be performed using sampling withuniformly progressing windows in one or more implementations.

In one or more implementations, the gesture prediction operationsdisclosed herein (e.g., using machine learning model 304) can bepersonalized, or tuned for a specific user. This personalized gestureprediction can be helpful, for example, to provide prediction and/ordetection of a gesture performed by a user who typically performs thegesture quickly (e.g., over a first period of time) and also for userswho typically perform the gesture slowly (e.g., over a second period oftime that is longer than the first period of time). This personalizedgesture prediction can also be helpful for prediction and/or detectionof a gesture as performed by different users with different physicalabilities, for prediction and/or detection of a gesture as indicated bydata generated with other static and/or dynamic user variability, forprediction and/or detection of a gesture that varies with movementvariability between users, and/or for prediction and/or detection of agesture generated by users with variations in device-wearing preferences(e.g., variations in band tightness for a smartwatch).

In one example of an implementation including personalized gesturerecognition, a device of a user can (e.g., during a gesture registrationprocess for the user and the device, and/or over time during use of thedevice by the user) build a library of known gestures for that user.Once a library of known gestures is available, the machine learningmodel may be modified and/or changed from a gestureprediction/recognition model to a gesture matching model, in which newinput sensor data is matched to corresponding signal data for one of thegestures in the gesture library, to identify the gesture beingperformed.

For example, a registration process may be performed for a user for thefirst time a user is interacting with a machine learning model forgesture prediction and/or recognition. For example, in one or moreimplementations, a device such as electronic device 102 may provide arequest to the user to perform one or more gestures of interest, andregister the performed gestures as their way of performing the gesture.A machine learning model such as machine learning model 304 may then usethese registered user-specific gestures as training data for betteridentifying specific types of gestures for that specific user. In thisway, the user can customize the gestures according to the way (e.g., thespeed or any physical abilities or preferences) the user is comfortableperforming the gestures, and the gesture prediction/recognition modelcan adapt to the user's behavior.

In another example of personalized gesture recognition, personalizedfederated learning operations can be performed to train and/or to tuneor personalize a machine learning model to identify or predict gesturesperformed by a particular user.

For example, in one or more implementations, a machine learning modelsuch as machine learning model 304 may utilize the federated learningtechnique to train and/or refine the model across multiple decentralizeddevices holding local samples, without exchanging samples or aggregatingmultiple model updates from decentralized mobile devices. In this way,multiple users can contribute to training a common model, whilepreserving the privacy of the users by avoiding sharing user informationbetween users.

In one or more implementations, a machine learning model can be trainedusing a federated learning technique to obtain a common initial modeltrained in the manner described above, and can then be further trainedlocally at the user's device to be customized to a specific user (e.g.,using a gesture registration process or sample data from the specificuser and device for model personalization).

FIG. 9 illustrates a flow diagram of example process formachine-learning based gesture recognition in accordance with one ormore implementations. For explanatory purposes, the process 900 isprimarily described herein with reference to the electronic device 102of FIG. 1. However, the process 900 is not limited to the electronicdevice 102 of FIG. 1, and one or more blocks (or operations) of theprocess 900 may be performed by one or more other components and othersuitable devices (e.g., any of the electronic devices 102-104). Furtherfor explanatory purposes, the blocks of the process 900 are describedherein as occurring in serial, or linearly. However, multiple blocks ofthe process 900 may occur in parallel. In addition, the blocks of theprocess 900 need not be performed in the order shown and/or one or moreblocks of the process 900 need not be performed and/or can be replacedby other operations.

At block 902, sensor data may be received from a sensor of a device suchas electronic device 102. The sensor data may include biosignalsensor(s) 206 such as from a photoplethysmography (PPG) sensor. The PPGsensor may include at least one of an infrared light source or a colorlight source. In one or more implementations, the first sensor outputmay indicate a change in blood flow. The sensor data may include sensordata from one or more of the non-biosignal sensor(s) 208. Thenon-biosignal sensor(s) 208 may be an accelerometer and/or a microphone,for example. Receiving the sensor data may be based on a determination(e.g., by a mode detection head of the machine learning model) that thedevice is in a gesture detection mode. Receiving the sensor data mayinclude receiving the sensor data during a first window of time that atleast partially overlaps a gesture time (e.g., gesture duration 606) ofthe gesture. Additional sensor data from the sensor of the device mayalso be received during one or more additional windows of time such as asecond window of time that at least partially overlaps the gesture timeof the gesture.

At block 904, the sensor data may be provided as input to a machinelearning model (e.g., machine learning model 304), the machine learningmodel having been trained to output, while a gesture is being performedby a user of the device and prior to completion of the gesture, apredicted gesture, a predicted start time (e.g., start time 608) of thegesture, and a predicted end time (e.g., end time 610) of the gesture,based on the sensor data. In one or more implementations, additionalsensor data (e.g., from the second window of time and/or one or moreadditional windows of time) may also be provided as input to the machinelearning model. In one or more implementations, the machine learningmodel may have been trained to output the predicted start time of thegesture and the predicted end time of the gesture at least in part bygenerating a gesture label such as gesture label 700 and a no-gesturelabel such as no-gesture label 702 for each of multiple windows of time(e.g., as described above in connection with FIGS. 7 and 8). Forexample, the gesture label and the no-gesture label may each have avalue that is smoothly continuous (e.g., linearly continuous,exponentially continuous, sigmoid continuous, or otherwise continuous)between a maximum value and a minimum value.

At block 906, the predicted gesture may be determined based on an outputfrom the machine learning model. In one or more implementations,determining the predicted gesture based on the output from the machinelearning model may include determining the predicted gesture based onthe output from the model that is based on the sensor data from thefirst window of time and based on an additional output of the machinelearning model that is based on the additional sensor data from thesecond window of time. Determining the predicted gesture based on theoutput from the machine learning model that is based on the sensor datafrom the first window of time and the additional output of the machinelearning model that is based on the additional sensor data from thesecond window of time may include aggregating a first predicted starttime from the machine learning model that is based on the sensor datafrom the first window of time and a second predicted start time from themachine learning model that is based on the additional sensor data fromthe second window of time to determine a final predicted start time forthe gesture. Determining the predicted gesture based on the output fromthe machine learning model that is based on the sensor data from thefirst window of time and the additional output of the machine learningmodel that is based on the additional sensor data from the second windowof time may include aggregating a first predicted end time from themachine learning model that is based on the sensor data from the firstwindow of time and a second predicted end time from the machine learningmodel that is based on the additional sensor data from the second windowof time to determine a final predicted end time for the gesture. In oneor more implementations, a size of an input buffer for the machinelearning model may be adjusted (e.g., increased or decreased) based onthe final predicted start time and the final predicted end time (e.g.,to include all of the sensor data between the final predicted start timeand the final predicted end time corresponding to the data for theentire gesture). Determining the predicted gesture may includedetermining the predicted gesture at a time after the final predictedend time using sensor data in the input buffer having the adjusted size.

At block 908, in response to determining the predicted gesture, apredetermined action may be performed on the device.

As described above, one aspect of the present technology is thegathering and use of data available from specific and legitimate sourcesfor gesture recognition. The present disclosure contemplates that insome instances, this gathered data may include personal information datathat uniquely identifies or can be used to identify a specific person.Such personal information data can include demographic data,location-based data, online identifiers, telephone numbers, emailaddresses, home addresses, data or records relating to a user's healthor level of fitness (e.g., vital signs measurements, medicationinformation, exercise information), date of birth, or any other personalinformation.

The present disclosure recognizes that the use of such personalinformation data, in the present technology, can be used to the benefitof users. For example, the personal information data can be used forgesture recognition. Accordingly, use of such personal information datamay facilitate transactions (e.g., on-line transactions). Further, otheruses for personal information data that benefit the user are alsocontemplated by the present disclosure. For instance, health and fitnessdata may be used, in accordance with the user's preferences to provideinsights into their general wellness, or may be used as positivefeedback to individuals using technology to pursue wellness goals.

The present disclosure contemplates that those entities responsible forthe collection, analysis, disclosure, transfer, storage, or other use ofsuch personal information data will comply with well-established privacypolicies and/or privacy practices. In particular, such entities would beexpected to implement and consistently apply privacy practices that aregenerally recognized as meeting or exceeding industry or governmentalrequirements for maintaining the privacy of users. Such informationregarding the use of personal data should be prominently and easilyaccessible by users, and should be updated as the collection and/or useof data changes. Personal information from users should be collected forlegitimate uses only. Further, such collection/sharing should occur onlyafter receiving the consent of the users or other legitimate basisspecified in applicable law. Additionally, such entities should considertaking any needed steps for safeguarding and securing access to suchpersonal information data and ensuring that others with access to thepersonal information data adhere to their privacy policies andprocedures. Further, such entities can subject themselves to evaluationby third parties to certify their adherence to widely accepted privacypolicies and practices. In addition, policies and practices should beadapted for the particular types of personal information data beingcollected and/or accessed and adapted to applicable laws and standards,including jurisdiction-specific considerations which may serve to imposea higher standard. For instance, in the US, collection of or access tocertain health data may be governed by federal and/or state laws, suchas the Health Insurance Portability and Accountability Act (HIPAA);whereas health data in other countries may be subject to otherregulations and policies and should be handled accordingly.

Despite the foregoing, the present disclosure also contemplatesembodiments in which users selectively block the use of, or access to,personal information data. That is, the present disclosure contemplatesthat hardware and/or software elements can be provided to prevent orblock access to such personal information data. For example, in the caseof gesture recognition, the present technology can be configured toallow users to select to “opt in” or “opt out” of participation in thecollection of personal information data during registration for servicesor anytime thereafter. In addition to providing “opt in” and “opt out”options, the present disclosure contemplates providing notificationsrelating to the access or use of personal information. For instance, auser may be notified upon downloading an app that their personalinformation data will be accessed and then reminded again just beforepersonal information data is accessed by the app.

Moreover, it is the intent of the present disclosure that personalinformation data should be managed and handled in a way to minimizerisks of unintentional or unauthorized access or use. Risk can beminimized by limiting the collection of data and deleting data once itis no longer needed. In addition, and when applicable, including incertain health related applications, data de-identification can be usedto protect a user's privacy. De-identification may be facilitated, whenappropriate, by removing identifiers, controlling the amount orspecificity of data stored (e.g., collecting location data at city levelrather than at an address level), controlling how data is stored (e.g.,aggregating data across users), and/or other methods such asdifferential privacy.

Therefore, although the present disclosure broadly covers use ofpersonal information data to implement one or more various disclosedembodiments, the present disclosure also contemplates that the variousembodiments can also be implemented without the need for accessing suchpersonal information data. That is, the various embodiments of thepresent technology are not rendered inoperable due to the lack of all ora portion of such personal information data.

FIG. 10 illustrates an electronic system 1000 with which one or moreimplementations of the subject technology may be implemented. Theelectronic system 1000 can be, and/or can be a part of, one or more ofthe electronic devices 102-104, and/or one or the server 108 shown inFIG. 1. The electronic system 1000 may include various types of computerreadable media and interfaces for various other types of computerreadable media. The electronic system 1000 includes a bus 1008, one ormore processing unit(s) 1012, a system memory 1004 (and/or buffer), aROM 1010, a permanent storage device 1002, an input device interface1014, an output device interface 1006, and one or more networkinterfaces 1016, or subsets and variations thereof.

The bus 1008 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of theelectronic system 1000. In one or more implementations, the bus 1008communicatively connects the one or more processing unit(s) 1012 withthe ROM 1010, the system memory 1004, and the permanent storage device1002. From these various memory units, the one or more processingunit(s) 1012 retrieves instructions to execute and data to process inorder to execute the processes of the subject disclosure. The one ormore processing unit(s) 1012 can be a single processor or a multi-coreprocessor in different implementations.

The ROM 1010 stores static data and instructions that are needed by theone or more processing unit(s) 1012 and other modules of the electronicsystem 1000. The permanent storage device 1002, on the other hand, maybe a read-and-write memory device. The permanent storage device 1002 maybe a non-volatile memory unit that stores instructions and data evenwhen the electronic system 1000 is off. In one or more implementations,a mass-storage device (such as a magnetic or optical disk and itscorresponding disk drive) may be used as the permanent storage device1002.

In one or more implementations, a removable storage device (such as afloppy disk, flash drive, and its corresponding disk drive) may be usedas the permanent storage device 1002. Like the permanent storage device1002, the system memory 1004 may be a read-and-write memory device.However, unlike the permanent storage device 1002, the system memory1004 may be a volatile read-and-write memory, such as random accessmemory. The system memory 1004 may store any of the instructions anddata that one or more processing unit(s) 1012 may need at runtime. Inone or more implementations, the processes of the subject disclosure arestored in the system memory 1004, the permanent storage device 1002,and/or the ROM 1010. From these various memory units, the one or moreprocessing unit(s) 1012 retrieves instructions to execute and data toprocess in order to execute the processes of one or moreimplementations.

The bus 1008 also connects to the input and output device interfaces1014 and 1006. The input device interface 1014 enables a user tocommunicate information and select commands to the electronic system1000. Input devices that may be used with the input device interface1014 may include, for example, alphanumeric keyboards and pointingdevices (also called “cursor control devices”). The output deviceinterface 1006 may enable, for example, the display of images generatedby electronic system 1000. Output devices that may be used with theoutput device interface 1006 may include, for example, printers anddisplay devices, such as a liquid crystal display (LCD), a lightemitting diode (LED) display, an organic light emitting diode (OLED)display, a flexible display, a flat panel display, a solid statedisplay, a projector, or any other device for outputting information.One or more implementations may include devices that function as bothinput and output devices, such as a touchscreen. In theseimplementations, feedback provided to the user can be any form ofsensory feedback, such as visual feedback, auditory feedback, or tactilefeedback; and input from the user can be received in any form, includingacoustic, speech, or tactile input.

Finally, as shown in FIG. 10, the bus 1008 also couples the electronicsystem 1000 to one or more networks and/or to one or more network nodes,such as the server 108 shown in FIG. 1, through the one or more networkinterface(s) 1016. In this manner, the electronic system 1000 can be apart of a network of computers (such as a LAN, a wide area network(“WAN”), or an Intranet, or a network of networks, such as the Internet.Any or all components of the electronic system 1000 can be used inconjunction with the subject disclosure.

Implementations within the scope of the present disclosure can bepartially or entirely realized using a tangible computer-readablestorage medium (or multiple tangible computer-readable storage media ofone or more types) encoding one or more instructions. The tangiblecomputer-readable storage medium also can be non-transitory in nature.

The computer-readable storage medium can be any storage medium that canbe read, written, or otherwise accessed by a general purpose or specialpurpose computing device, including any processing electronics and/orprocessing circuitry capable of executing instructions. For example,without limitation, the computer-readable medium can include anyvolatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM,and TTRAM. The computer-readable medium also can include anynon-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM,NVRAM, flash, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM,NRAM, racetrack memory, FJG, and Millipede memory.

Further, the computer-readable storage medium can include anynon-semiconductor memory, such as optical disk storage, magnetic diskstorage, magnetic tape, other magnetic storage devices, or any othermedium capable of storing one or more instructions. In one or moreimplementations, the tangible computer-readable storage medium can bedirectly coupled to a computing device, while in other implementations,the tangible computer-readable storage medium can be indirectly coupledto a computing device, e.g., via one or more wired connections, one ormore wireless connections, or any combination thereof.

Instructions can be directly executable or can be used to developexecutable instructions. For example, instructions can be realized asexecutable or non-executable machine code or as instructions in ahigh-level language that can be compiled to produce executable ornon-executable machine code. Further, instructions also can be realizedas or can include data. Computer-executable instructions also can beorganized in any format, including routines, subroutines, programs, datastructures, objects, modules, applications, applets, functions, etc. Asrecognized by those of skill in the art, details including, but notlimited to, the number, structure, sequence, and organization ofinstructions can vary significantly without varying the underlyinglogic, function, processing, and output.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, one or more implementationsare performed by one or more integrated circuits, such as ASICs orFPGAs. In one or more implementations, such integrated circuits executeinstructions that are stored on the circuit itself.

Those of skill in the art would appreciate that the various illustrativeblocks, modules, elements, components, methods, and algorithms describedherein may be implemented as electronic hardware, computer software, orcombinations of both. To illustrate this interchangeability of hardwareand software, various illustrative blocks, modules, elements,components, methods, and algorithms have been described above generallyin terms of their functionality. Whether such functionality isimplemented as hardware or software depends upon the particularapplication and design constraints imposed on the overall system.Skilled artisans may implement the described functionality in varyingways for each particular application. Various components and blocks maybe arranged differently (e.g., arranged in a different order, orpartitioned in a different way) all without departing from the scope ofthe subject technology.

It is understood that any specific order or hierarchy of blocks in theprocesses disclosed is an illustration of example approaches. Based upondesign preferences, it is understood that the specific order orhierarchy of blocks in the processes may be rearranged, or that allillustrated blocks be performed. Any of the blocks may be performedsimultaneously. In one or more implementations, multitasking andparallel processing may be advantageous. Moreover, the separation ofvarious system components in the implementations described above shouldnot be understood as requiring such separation in all implementations,and it should be understood that the described program components andsystems can generally be integrated together in a single softwareproduct or packaged into multiple software products.

As used in this specification and any claims of this application, theterms “base station”, “receiver”, “computer”, “server”, “processor”, and“memory” all refer to electronic or other technological devices. Theseterms exclude people or groups of people. For the purposes of thespecification, the terms “display” or “displaying” means displaying onan electronic device.

As used herein, the phrase “at least one of” preceding a series ofitems, with the term “and” or “or” to separate any of the items,modifies the list as a whole, rather than each member of the list (i.e.,each item). The phrase “at least one of” does not require selection ofat least one of each item listed; rather, the phrase allows a meaningthat includes at least one of any one of the items, and/or at least oneof any combination of the items, and/or at least one of each of theitems. By way of example, the phrases “at least one of A, B, and C” or“at least one of A, B, or C” each refer to only A, only B, or only C;any combination of A, B, and C; and/or at least one of each of A, B, andC.

The predicate words “configured to”, “operable to”, and “programmed to”do not imply any particular tangible or intangible modification of asubject, but, rather, are intended to be used interchangeably. In one ormore implementations, a processor configured to monitor and control anoperation or a component may also mean the processor being programmed tomonitor and control the operation or the processor being operable tomonitor and control the operation. Likewise, a processor configured toexecute code can be construed as a processor programmed to execute codeor operable to execute code.

Phrases such as an aspect, the aspect, another aspect, some aspects, oneor more aspects, an implementation, the implementation, anotherimplementation, some implementations, one or more implementations, anembodiment, the embodiment, another embodiment, some implementations,one or more implementations, a configuration, the configuration, anotherconfiguration, some configurations, one or more configurations, thesubject technology, the disclosure, the present disclosure, othervariations thereof and alike are for convenience and do not imply that adisclosure relating to such phrase(s) is essential to the subjecttechnology or that such disclosure applies to all configurations of thesubject technology. A disclosure relating to such phrase(s) may apply toall configurations, or one or more configurations. A disclosure relatingto such phrase(s) may provide one or more examples. A phrase such as anaspect or some aspects may refer to one or more aspects and vice versa,and this applies similarly to other foregoing phrases.

The word “exemplary” is used herein to mean “serving as an example,instance, or illustration”. Any embodiment described herein as“exemplary” or as an “example” is not necessarily to be construed aspreferred or advantageous over other implementations. Furthermore, tothe extent that the term “include”, “have”, or the like is used in thedescription or the claims, such term is intended to be inclusive in amanner similar to the term “comprise” as “comprise” is interpreted whenemployed as a transitional word in a claim.

All structural and functional equivalents to the elements of the variousaspects described throughout this disclosure that are known or latercome to be known to those of ordinary skill in the art are expresslyincorporated herein by reference and are intended to be encompassed bythe claims. Moreover, nothing disclosed herein is intended to bededicated to the public regardless of whether such disclosure isexplicitly recited in the claims. No claim element is to be construedunder the provisions of 35 U.S.C. § 112(f) unless the element isexpressly recited using the phrase “means for” or, in the case of amethod claim, the element is recited using the phrase “step for”.

The previous description is provided to enable any person skilled in theart to practice the various aspects described herein. Variousmodifications to these aspects will be readily apparent to those skilledin the art, and the generic principles defined herein may be applied toother aspects. Thus, the claims are not intended to be limited to theaspects shown herein, but are to be accorded the full scope consistentwith the language claims, wherein reference to an element in thesingular is not intended to mean “one and only one” unless specificallyso stated, but rather “one or more”. Unless specifically statedotherwise, the term “some” refers to one or more. Pronouns in themasculine (e.g., his) include the feminine and neuter gender (e.g., herand its) and vice versa. Headings and subheadings, if any, are used forconvenience only and do not limit the subject disclosure.

1. A method comprising: receiving, from a first sensor of a device,first sensor output of a first type; receiving, from a second sensor ofthe device, second sensor output of a second type that differs from thefirst type; providing the first sensor output and the second sensoroutput as inputs to a machine learning model, the machine learning modelhaving been trained to output a predicted gesture based on sensor outputof the first type and sensor output of the second type; determining thepredicted gesture based on an output from the machine learning model;and performing, in response to determining the predicted gesture, apredetermined action on the device.
 2. The method of claim 1, whereinthe first sensor of the device comprises a photoplethysmography (PPG)sensor.
 3. The method of claim 2, wherein the PPG sensor comprises atleast one of an infrared light source or a color light source.
 4. Themethod of claim 2, wherein the first sensor output indicates a change inblood flow.
 5. The method of claim 1, wherein the second sensor of thedevice comprises at least one of an accelerometer or a microphone. 6.The method of claim 1, wherein at least one of receiving the firstsensor output or receiving the second sensor output is based on adetermination that the device is in a gesture detection mode.
 7. Themethod of claim 1, the machine learning model having been trained acrossmultiple different users.
 8. The method of claim 1, wherein thepredicted gesture comprises at least one of a finger-based gesture, or awrist-based gesture.
 9. The method of claim 8, wherein the finger-basedgesture comprises at least one of a finger pinch gesture or afist-clinch gesture.
 10. The method of claim 1, wherein thepredetermined action corresponds to changing a user interface on thedevice.
 11. A device, comprising: a first sensor; a second sensor; atleast one processor; and a memory including instructions that, whenexecuted by the at least one processor, cause the at least one processorto: receive, from the first sensor, first sensor output of a first type;receive, from the second sensor, second sensor output of a second typethat differs from the first type; provide the first sensor output andthe second sensor output as inputs to a machine learning model, themachine learning model having been trained to output a predicted gesturebased on sensor output of the first type and sensor output of the secondtype; determine the predicted gesture based on an output from themachine learning model; and perform, in response to determining thepredicted gesture, a predetermined action on the device.
 12. The deviceof claim 11, wherein the first sensor comprises a photoplethysmography(PPG) sensor.
 13. The device of claim 12, wherein the PPG sensorcomprises at least one of an infrared light source or a color lightsource.
 14. The device of claim 12, wherein the first sensor outputindicates a change in blood flow.
 15. The device of claim 11, whereinthe second sensor the device comprises at least one of an accelerometeror a microphone.
 16. The device of claim 11, wherein at least one ofreceiving the first sensor output or receiving the second sensor outputis based on a determination that the device is in a gesture detectionmode.
 17. The device of claim 11, the machine learning model having beentrained across a general population of users.
 18. The device of claim11, wherein the predicted gesture comprises at least one of afinger-based gesture, or a wrist-based gesture.
 19. The device of claim18, wherein the finger-based gesture comprises at least one of a fingerpinch gesture or a fist-clinch gesture.
 20. A computer program productcomprising code, stored in a non-transitory computer-readable storagemedium, the code comprising: code to receive, from a first sensor of adevice, first sensor output of a first type, the first sensor comprisinga bio-signal sensor; code to receive, from a second sensor of thedevice, second sensor output of a second type that differs from thefirst type; code to provide the first sensor output and the secondsensor output as inputs to a machine learning model, the machinelearning model having been trained to output a predicted gesture basedon sensor output of the first type and sensor output of the second type;code to determine the predicted gesture based on an output from themachine learning model; and code to perform, in response to determiningthe predicted gesture, a predetermined action on the device.
 21. Thecomputer program product of claim 20, the code further comprising: codeto perform a gesture registration processes for a user of the device;and code to tune the machine learning model for the user based on thegesture registration process. 22-28. (canceled)